feat(connectors): Delta Lake Sink Connector#2889
feat(connectors): Delta Lake Sink Connector#2889kriti-sc wants to merge 25 commits intoapache:masterfrom
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2889 +/- ##
============================================
+ Coverage 70.66% 70.81% +0.15%
Complexity 943 943
============================================
Files 1114 1118 +4
Lines 94778 95515 +737
Branches 71978 72732 +754
============================================
+ Hits 66976 67643 +667
- Misses 25332 25380 +48
- Partials 2470 2492 +22
🚀 New features to boost your workflow:
|
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR. Thank you for your contribution! |
| let mut opts = HashMap::new(); | ||
|
|
||
| match config.storage_backend_type.as_deref() { | ||
| Some("s3") => { |
There was a problem hiding this comment.
Let's use custom enum to avoid magic strings.
|
|
||
| pub(crate) fn build_storage_options( | ||
| config: &DeltaSinkConfig, | ||
| ) -> Result<HashMap<String, String>, Error> { |
There was a problem hiding this comment.
Maybe instead of returning a hash map, we could have the dedicated structs (per storage provider) with optional fields - and then either an enum could be returned or a single struct with all available storages as optional fields?
| ), | ||
| ]; | ||
| let storage_options = HashMap::from([ | ||
| ("AWS_ACCESS_KEY_ID".into(), MINIO_ACCESS_KEY.into()), |
There was a problem hiding this comment.
Same here (based on my previous suggestion) - we would use the dedicated option struct, and then simply serialize it.
5 participants
Which issue does this PR close?
Closes #1852
Rationale
Delta Lake is a data analytics engine, and very popular in modern streaming analytics architectures.
What changed?
Introduces a Delta Lake Sink Connector that enables writing data from Iggy to Delta Lake.
The Delta Lake writing logic is heavily inspired by the kafka-delta-ingest project, to have a proven starting ground for writing to Delta Lake.
Local Execution
user_id: String, user_type: u8, email: String, source: String, state: String, created_at: DateTime<Utc>, message: Stringusing sample data producer.AI Usage
If AI tools were used, please answer: